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ABSTRACT 

Often a research department in a school system is 
called on to make an after the fact evaluation of a program or 
project. Although the department is operating under a handicap, it 
can still provide some data useful for evaluative purposes. It is 
suggested that all the classical methods of descriptive statistics be 
brought into play. The use of these techniques in evaluating the 
effectiveness of a community advisory committee for a Board of 
Education is described in detail. (DG) 
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-A MODEL F0k POST HOC EVALUATION 



The public school researcher is frequently asked by a school \ 

I 

administrator to provide evaluative information about a project in which he has 

not been involved and which is either near completion or completed. The researcher 

w 

has not been involved in the planning of the project nor in its execution, and yet 
he is expected to provide jiome kind of evaluation of the project for the decision 
makers, to provide them information upon which to base a decision relative to the 
project . 

An example of this is the case which I will cite throughout this paper. 
The Superintendent and the Board of Education agreed with a group of parents to 
develop a Community School Advisory Committee in one of the transitional areas of 
Philadelphia. A resolution was passed by the Board of Education making this group 
an. advisory committee to work with school officials in planning and developing a 
program for the new school under construction. A clause in the resolution stated 
that a written evaluation of the project was to be made to the Board ten months 
later which would decide the fate of the Committee. Unfortunately, no one in the 
Research Office was made aware of this resolution. Such instances are not unusual, 
however, and we have been asked in the past to evaluate summer school programs, 
ongoing curriculum programs and other kinds of instructional programs which have 

4 

been carried out in the schools and frequently completed before we are asked to 
provide any kind of research help. 

s * I 

We have made a concerted effort in Philadelphia to educate the adminis- 
trators of the necessity for our being involved in the early phases of planning 
of projects if we are to answer the really significant questions which must be 
answered about the efficacy of various projects of this nature. Much progress 
has been made and we are now involved in the planning stages of most projects of 
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any magnitude in the System, and are only occasionally given a project such as 
the one described in this paper. However, it is still true, and will continue to 
be true, that school district researchers will be faced with the problem of 
conducting post hoc studies because of the nature of man and the nature of 
communication in the large bureaucratic organizations which are the schools today. 

It has been our experience that newly trained researchers are prone to 

say to school administrators, when faced with this kind of problem, "Well, since 

» 

I wasn't involved in the planning, I cannot answer the questions you want answered." 

it 

Their training has emphasized statistical evaluations using ANOVA or Covariance 

j 

Analyses, Multivariate Analyses or Correlational Analyses to such an extent that 

• they do not feel comfortable using less sophisticated techniques. Unfortunately, 

this kind of response frequently "turns off" administrators toward research, and 

researchers are excluded from the planning of future projects as a result. Also 

when the dollars are hard to find, researchers find themselves being cut from 

* 

budgets. For these reasons, it is politic to be able to find some way of answering 

these questions whenever possible. 

A second question frequently posed by young, newly trained research 
personnel is, "Aren't you 'watering down' your research if you do this?" I re- 
phrase the question and ask, "Do we have to have a great deal of information before 
we can say anything about a project or can we respond in terms of limited information? 
Another way of stating it is, "is it better to make decisions based on some infor- 
mation or no information?" Since we were not involved in the planning of these 

' 

I projects, obviously we cannot answer all of the questions that we would like to 

answer. It is imperative t6 control many variables if specific answers are to be 
obtained for specific questions. V/e feel that if we can provide some information, 
however, it is better for the decision maker than to have no information at all upon 



which to base his decision. 
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With this as our rationale, we reviewed the Board Resolution which 
instituted the project that we were asked to evaluate to determine what information 



would be needed by the Board. We found that there was a statement of objectives 
but as is typical of most studies in which researchers are not involved, the 
objectives were so loosely stated as to be very difficult to interpret. There was 
a list of activities which had been agreed to as necessary for the Committee to 



perform if it was to accomplish these goals, but there was no statement uf any 

i 

criteria against which these activities could be measured or the objectives evaluated. 

I 

\ ( Tfle first order of' business then was to determine precisely what the 
objectives of this project were. The Board of Education had approved a project and 
it had been carried out by a group of public spirited, community people, in coopera- 
tion with school district personnel. 



The resolution developed from a philosophy which 
Mario Fantinni has stated very well. I would like 
to quote from Gittell's Participants and 
Participation at this point to indicate the kjnd of 
goals that the Committee had set for itself in this 
respect. "The people still pay for public education 
and it is from them that the schools draw continuing 
sanction, but it is a myth that sanction and support 
add up to control. 

The reality is that control of public education in 
our large cities has passed over almost exclusively 
to management. Large city school systems have taken 
the shape of massive corporate enterprises, increasing jy 
distant from the public. Any effort to change the school 
system and expand civic participation must face the con- 
centration of power in the professional bureaucracy and 
the resistance by the bureaucracy to any plan that would 
erode its power." 

It seemed to us that it would be logical to determine the perception of these 

objectives which grew out of this philosophy from the point of view of each of 
the groups involved. 



Hsiang descrtpttye statCsfi.es., historical or content research, observations, 
and surveys, m began to make an assessment of the perceptions of the people based 

k * 

on their spoken or written words. ' 
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j 

First, all written documents, including minutes of every meeting, u^re read 
thoroughly and activities tabulated. Next, each committee member was interviewed, 

I 

and school district personnel having direct contact with the committee people were 
interviewed. In each case, the groups were asked what they felt were the objectives 
of the Board of Education when it formed this committee. Based on these results, we 
were able to formulate a clear statement of the perceptions of the objectives in 
specific terms. Exceptions to the generally agreed upon statement of objectives 
were noted, and included in the report to the Board. 

Once the objectives had been determined, then an analysis was made of the 
actual behaviors of the people involved in the project; that is, the Board of Educa- 
tion in setting Up the project had outlined duties and responsibilities which this 
Committee was to perform during the year. Once again, minutes of the meetings, 
materials written by the principal of the yet to be built school, and other school 
district personnel were examined to determine how often and to what degree, each of 
these responsibilities had been carried out by the Committee. 

The frequency of attendance at meetings, the number of hours spent in 
meetings, the topics discussed, and the actions taken by the Committee were listed 
in tables for clear understanding. The tables were set up so that comparisons 
between perceptions of Committee members, school district personnel and the com- 
munity groups who had been interviewed also could be made. Judgments as to how 
effectively these duties had been carried out by the Committee could then be made 
by the Board of Education. 

Just as the objectives of the project and the activities prescribed by 
the Board of Education had not been clearly stated, and required clarification, so 
the criteria by which the Committee was to be evaluated had to be established. To 
establish criteria, each of the Committee members and each of the school administrators 
was asked how he would rate the Committee in terms of having carried out the functions 
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assigned him by the Board of Education. Each was then asked to give specific in- 
stances of important accomplishments made during the preceding ten months as well 
as specific instances of failures. On the basis of these specific bahavioral state- 
ments, judgments could then be made by the Board as to how far and to what extent 
the Committee had actually met the requirements as they were perceived by those 
involved. 

The actual criterion against which the Committee was judged was the 

accomplishment of goals which they had set for themselves in carrying out the 

I 

| • ^ 
activities prescribed by the Board in the judgment of the Board of Education itself. 

This plan follows the kind of analysis suggested by Stake in his judgmental model 

for evaluation. The measure of the effectiveness of the Committee in working 

within itself and with school district personnel was not difficult to determine. The 

Committee had been very active and had accomplished most of the goals which had been 

set. A set of goals which the Board had prescribed, however, dealing with their 

% 

participation in the community required additional information. 

To determine how the Community perceived this Committee as having been 
representative of their views, a local professional opinion research group was hired 
to conduct interviews with a stratified sample of local community people. Because 
the community was integrated but predominantly Black, the opinion research group 

sent largely Black interviewers into the community to elict responses to questions 

\ ■ !' 

I 1 

| about how they perceived the Committee, its work and its accomplishment. Since there 

V. 

is evidence that the race of the interviewer does affect the responses obtained,, 
wherever possible interviewers and respondents were of the same race. In addition, 

V ' 

Iji sample of local businessmen was interviewed by school district personnel. On the 
best's of this sampling, it wus determined that the Committee had not been effective 

i n \meet i ng its goal of communicating wi th the community about the problems of the 

% . * , 

school . One of the arguments which the Committee had used to establish itself was 
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was the need for greater community participation in the schools. With this phil- 
osophy as one basis for their existence, the Committee was expected to have made 
an impact on the community and to have brought about increased participation by com- 
munity people. The surveys that we conducted indicated that most people were not 
aware of the Committee's existence. In addition, attendance at local school meetings 
and the quarterly meetings called by this Committee was also very poor. This infor- 
mation was included in the report to the Board of Education and the data made availabl 

to the Committee. 

i 

4 i t 

j On the basis of these analyses, utilizing only post hoc research techniques, 

we were able to provide information to the Board of Education upon which they could 

make their decision about the Committee's continued existence. The conclusion of 

the Board was that the Committee should remain as an advisory group to the school 

since it had accomplished most administration of the goals established for it, but 

that they should concentrate their efforts toward obtaining greater community 

% 

participation during the coming year. The research office was required to participate 
actively in the project throughout the coming year. 

One of the interesting serendipi tious results of the study was that the 
Committee, which had initially been very resistive to a Board of Education researcher, 
evaluating their activities, found the information provided so helpful to them in 
making their plans for the coming year that they accorded the research team every 
kind of assistance in carrying out the research. 

V 

The purpose of this paper was to demonstrate how classic descriptive 
statistical techniques can be used in modern day education to answer significant 
questions raised by school district administrators. 

All of the classical kinds of descriptive research techniques were used 

/ 

in this study except correlation. These techniques provided meaningful information 
to the Board of Education which allowed them to make tbeir decision about renewing 
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the contract with the community group on a rational objective basis, and permitted 
the Research Office to become more actively involved in the planning and evaluation 

4 

of the project in its second year of operation. 



WCT : rm 

March 1 1 , 1970 
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